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Abstract 

This paper is concerned with two-block separable convex minimization prob¬ 
lems with linear constraints, for which it is either impossible or too expensive to 
obtain the exact solutions of the subproblems involved in the proximal ADMM (al¬ 
ternating direction method of multipliers). Such structured convex minimization 
problems often arise from the two-block regroup of three or four-block separable 
convex optimization problems with linear constraints, or from the constrained total- 
variation superresolution image reconstruction problems in image processing. For 
them, we propose an inexact indefinite proximal ADMM of step-size r £ (0, ) 

with two easily implementable inexactness criteria to control the solution accuracy 
of subproblems, and establish the convergence under a mild assumption on indefinite 
proximal terms. We apply the proposed inexact indefinite proximal ADMMs to the 
three or four-block separable convex minimization problems with linear constraints, 
which come from the important class of doubly nonnegative semidefinite program¬ 
ming (DNNSDP) problems with many linear equality and/or inequality constraints. 
Numerical results indicate that the inexact indefinite proximal ADMM with the 
absolute error criterion has a comparable performance with the directly extended 
multi-block ADMM of step-size r = 1.618 without convergence guarantee, whether 
in terms of the number of iterations or the computation time. 

Keywords: Separable convex optimization, inexact proximal ADMM, DNNSDPs 


1 Introduction 

Let X, Y and Z be the finite dimensional vector spaces endowed with the inner product 
(•, •) and its induced norm || • ||. Given closed proper convex functions / : X —)• (—oo, -|-oo] 
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and ( 7 : Y—)> (— 00 , + 00 ], we are concerned with the separable convex optimization problem 


min 




f{x)+g{y) 


s.t. A*X + B*y = c, 


( 1 ) 


where ^: Z —>• X and : Z —^ Y are the given linear operators, A* and B* denote the 
adjoint operators of A and B, respectively, and c G Z is a given vector. 

As well known, there are many important cases with the form of (1), which include 
the covariance selection problems and semidefinite least squares problems in statistics [ 1 , 
30, 39], the sparse plus low-rank recovery problem arising from the so-called robust PC A 
(principle component analysis) with noisy and incomplete data [34, 32], the constrained 
total-variation image restoration and reconstruction problems [22, 29], the simultaneous 
minimization of the nuclear norm and £i-norm of a matrix arising from the low-rank and 
sparse representation for image classification and subspace clustering [40, 36], and so on. 

For the structured convex minimization problem (1), the alternating direction method 
of multipliers (ADMM for short), first proposed by Glowinski and Marrocco [11] and 
Gabay and Mercier [12], is one of the most popular methods. For any given cr > 0, let 
Lo-: X X Y X Z —)■ (—00, -|-oo] denote the augmented Lagrangian function of problem (1) 

La{x,y,z) := f{x) + g{y) + {z,A*x + B*y - c) + ^\\A*x + B*y - c|p. 

The ADMM, from an initial point (x^, y^, z^) € dom / x dom 5 X Z, consists of the steps 


G argmin L(j{x,y^,z^), 

(2a) 

xex 


G argmin ,y, z'^), 

(2b) 

y&Y 


:^+i = z^ + Ta{A*x^+^ + B*y'^+^ - c ), 

(2c) 


where r G (0, is a constant to control the step-size in (2c). The iterative scheme 

of ADMM actually embeds a Gaussian-Seidel decomposition into each iteration of the 
classical augmented Lagraigan method of Hestenes-Powell-Rockafellar [14, 25, 28], so 
that the challenging task (i.e., the exact solution or the approximate solution with a high 
precision of the Lagrangian minimization problem) is relaxed to several easy ones. 

Notice that the subproblems (2a) and (2b) in the ADMM may have no closed-form 
solutions or even be difficult to solve. When the functions / and g enjoy a closed-form 
Moreau envelope, one usually introduces the proximal terms ^||x —and ^Hy — 
respectively into the subproblems ( 2 a) and ( 2 b) to cancel the operators AA* and BB* 
so as to get the exact solutions of proximal subproblems. This is the so-called proximal- 
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ADMM which, for a chosen initial point € dom/ X dom (7 X Z, consists of 



(3b) 


(3a) 


(3c) 


The existing works on the proximal ADMM mostly focus on the positive definite proximal 
terms (see, e.g., [15, 35, 41]). It is easy to see that the proximal subproblems with the 


positive definite proximal terms will have a big difference from the original subproblems 
of ADMM. In fact, as pointed out in the conclusion remarks of [15], “large and positive 
definite proximal terms will lead to easy solution of subproblems, but the number of 


iterations will increase. Therefore, for subproblems which are not extremely ill-posed, 
the proximal parameters should be small.” In view of this, some researchers recently 
develop the semi-proximal or indefinite proximal ADMM [38, 9, 21] by using the positive 
semidefinite even indefinite proximal terms. The numerical experiments in [9] show 
that such tighter proximal terms display better numerical performance. In addition, it is 
worthwhile to emphasize that the ADMM itself is a semi-proximal (of course an indefinite 
proximal) ADMM, but is not in the family of positive definite proximal ADMMs. 

In this paper we are interested in problem (1) in which the functions / and/or g may 
not have a closed-form Moreau envelope or the linear operators A and/or B have a large 
spectral norm (now the proximal subproblems with a positive definite proximal term 
are bad surrogates for those of the ADMM), for which it is impossible or too expensive 
to achieve the exact solutions of the proximal subproblems though they are unique. 
Such separable convex optimization problems arise directly from the constrained total- 
variation superresolution image reconstruction problems [4, 24] in image processing, and 
the two-block regroup of three or four-block separable convex minimization problems. 
Indeed, for the following four-block separable convex minimization problem 


nhn 

XiGXi 

S-t. = C 


(4) 


where /* : X* ^ (—oo,-|-oo] for i = 1,2 ,3,4 are closed proper convex functions, and 
: Z —>• Xj for i = 1, 2, 3,4 are linear operators, since the directly extended multi-block 
ADMM does not have the convergence guarantee (see the counterexamples in [3]), one 
may rearrange it as the form of (1) by reorganizing any two groups of variables into one 
group, and then apply the classical ADMM for solving the two-block regrouped problem. 
Clearly, the exact solution of each subproblem of ADMM for the two-block regrouped 
problem is difficult to obtain due to the cross of two classes of variables. In particular, 
the two-block regroup resolving of multi-block separable convex optimization also has a 
separate study value. 

To resolve this class of difficult two-block separable convex minimization problems, 
we propose an inexact indefinite proximal ADMM with a step-size r € (0, in which 
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the proximal subproblems are solved to a certain accuracy with two easily implementable 
inexactness criteria to control the accuracy. Here, an indefinite proximal term, instead 
of a positive definite proximal term, is introduced into each subproblem of the ADMM 
to guarantee that each proximal subproblem has a unique solution as well as becomes 
a good surrogate for the original subproblem of the ADMM. For the proposed inexact 
indefinite proximal ADMM, we establish its convergence under a mild assumption on 
the indefinite proximal terms. To the best of our knowledge, this is the first convergent 
inexact proximal ADMM in which step-size r may take the value in the interval (1, 

We notice that a few existing research papers on inexact versions of the ADMM all focus 
on the unit step-size; see [8, 15, 24, 13, 5], and moreover, only the papers [24, 13, 5] 
develop truly implementable inexactness criteria in the exact solutions are not required. 
Our inexact indefinite proximal ADMM is using the same absolute error criterion and 
and a little different relative error from the one used in [24]. It is well known that the 
ADMM with r = 1.618 requires less 20% to 50% iterations than the one with r = 1, 
especially for those difficult SDP problems [33]. Thus, the proposed inexact indefinite 
proximal ADMMs with a large step-size is expected to have better performance. 

In this work, we apply the inexact indefinite proximal ADMMs to the three and four- 
block separable convex minimization problems with linear constraints, coming from the 
duality of the doubly nonnegative semidefinite programming (DNNSDP) problems with 
many linear equality and/or inequality constraints. Specifically, we solve the two-block 
regroupment for the dual problems of DNNSDPs with the inexact indefinite proximal 
ADMM. Observe that the iterates yielded by solving each subproblem in an alternating 
way can satisfy the optimality condition approximately. Hence, in the implementation of 
the inexact indefinite proximal ADMMs, we get the inexact solution of each subproblem 
by minimizing the two group of variables alternately. Numerical results indicate that 
the inexact indefinite proximal ADMM with the absolute error criterion is comparable 
with the directly extended multi-block ADMM with step-size r = 1.618 whether in 
terms of the number of iterations or the computation time, while the one with the 
relative error criterion requires less outer-iterations but more computation time since the 
error criterion is more restrictive and requires more inner-iterations. Thus, the inexact 
indefinite proximal ADMM with the absolute error criterion provides an efficient tool for 
handling the three and four-block separable convex minimization problems. 

We observe that there are several recent works [37, 18, 19, 10] to regroup the multi¬ 
block separable convex minimization problems into two-block or several subblocks, and 
then solve each subblock simultaneously by introducing a positive definite proximal term 
related to the numbers of subproblems. Such procedures lead to easily solvable subprob¬ 
lems, but their performance becomes worse due to larger proximal terms. 

The rest of this paper is organized as follows. Section 2 gives some notations and the 
main assumption. Section 3 describes the inexact indefinite proximal ADMMs and ana¬ 
lyzes the properties of the sequence generated. The convergence of the inexact indefinite 
proximal ADMMs is established in Section 4. Section 5 applies the inexact indefinite 
proximal ADMMs for solving the duality of the doubly DNNSDPs with many linear 
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equality and/or inequality constraints. Some concluding remarks are given in Section 6. 


2 Notations and assumption 

Notice that the functions / : X —>■ (— 00 , + 00 ] and 5 : Y —>■ (— 00 , + 00 ] are closed proper 
convex, and the subdifferential mappings of closed proper convex functions are maximal 
monotone [26, Theorem 12.17]. Hence, there exist self-adjoint operators ^ 0 and 
Sg y 0 such that for all x,x G domf, u € df{x) and u € df{x), 

fix)>f{x) + {u,x — ^ + -\\x — x\\‘^^ and (n — n,x — x) > ||x — (5) 

and for all y,y G domg, v G dg{y) and v G dg{y), 

g{x) > g{y) + {v,y+ ^\\y-y\\l^ and {v -v,y - y) > \\y - yWl^^. ( 6 ) 

For a self-adjoint linear operator T: X —>■ X, the notation T Y 0 (respectively, T Y 0) 
means that T is positive semidefinite (respectively, positive definite), that is, (x, Tx) > 0 
for all X G X (respectively, (x,Tx) > 0 for all x G X\{0}). Given a self-adjoint positive 
semidefinite linear operator T: X —)• X, we denote by || • II 7 - the norm induced by T, i.e., 

IIxIIt" := (x,Tx) Vx G X. 

Given a self-adjoint positive definite linear operator, we denote by Amax(7~) and Amin(7~) 
the largest eigenvalue and the smallest eigenvalue of T, respectively, and by D-j-{x, H) the 
distance induced by T from x to a closed set H, that is, D-j-^x, H) := min^gfj \\z — x|| 7 -. 
When T is the identity operator, we suppress the notation T in D'f{x, H) and write 
simply D{x, H). Glearly, for any positive definite linear operator T: X —)• X and 7 > 0, 

2|(n,x)| < 7 “^||m||^- h 7 ||n||^_i Vn,xGX. (7) 

In addition, for any n, x G X and any self-adjoint linear operator 7”: X —)• X, the following 
two identities will be frequently used in the subsequent analysis: 

2{u,Tv) = {u,Tu) -I- {v,Tv) - {u - v,T{u - v)) 

= {u + v,T{u + v)) - {u,Tu) - {v,Tv). (8) 

Throughout this paper, we make the following assumption for problem (1): 

Assumption 2.1 Problem (1) has an optimal solution, to say [x*,y*) G dom/xdomg(, 
and there exists a point {x,y) G ri(dom/ X dom^r) such that A*x + B*y = b. 

Under Assumption 2.1, from [27, Gorollary 28.2.2 & 28.3.1] and [27, Theorem 6.5 & 
23.8], it follows that there exists a Lagrange multiplier z* G Z such that 

-Az* G dfix*), -Bz* G dg{x*) and A*x* + B*y* - c = 0 (9) 

where df and dg are the subdifferential mappings of / and g, respectively. Moreover, 
any 2 * G Z satisfying (9) is an optimal solution to the dual problem of (1). In the sequel, 
we call (x* ,y*, z*) G dom/ x domg' x Z a primal-dual solution pair of problem (1). 
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3 Inexact indefinite proximal ADMMs 

In this section, we describe the iteration steps of the inexact indefinite proximal ADMMs 
for solving problem (1), and then analyze the properties of the seqnence generated. 

The iteration steps of onr inexact indefinite proximal ADMMs are stated as follows. 

lEIDP-ADMM (Inexact indefinite proximal ADMM for (1)) 

(S.O) Let a,T > 0 be given. Choose self-adjoint linear operators : X —)■ X and 
Vg'.Y^Y snch that Tf '■= Vj YYif + aAA* >- 0 and Tg'.= Vg + T,g + aBB* >- 0. 
Choose an initial point , z^) £ dom/ X dom^ X Z. Set k := 0. 

(5.1) Find Ri argmin 2 ,gx (f>k{x) := L„{x, y^, z^) -|- |||x — 

(5.2) Find y^^^ « argmin^gy V’fc(y) := L„{x^+^,y,z^) + \\\y - y^\\p^. 

(5.3) Update the Lagrange mnltiplier z^~^^ via the following formnla 

^k+i + B*y^+^ - c). 

(5.4) Let k ^ k + 1, and go to Step (S.l). 


The approximate optimality in (S.l) and (S.2) is measnred by the following criteria: 
(Cl) D[^,d(j)k{x^^^)) <yk+i, D{p,dipk{y^^^)) <k>k+i and max(/rfc+i, i/^+i) < oo; 

(C2) Djr{Q,d(l)k{x^^^))<yk+i\\x^"^^-x^WTf, L»g;(0, dV'fc(2/''+^)) y^ll-TJ and 

^^0 ™ax(^fc+i, i/fc+i) < oo, where T" : X —)■ X and ^ : Y —>• Y are self-adjoint 
positive definite linear operators with Y and Q~^ Y 7^] 

(C2’) Djr[0,d(j)k{x^^^))<yk+i\\x^^^-x^\\rj>, Dg{0,d'ipkiy^^^)) <i^k+i\\y^^^ - y^\\Tg and 
YlT=o^^^(f^k+i^ ^k+i) ^ where T and Q are same as the one in (C2). 


Notice that (Cl) is an absolnte error criterion, while (C2) and (C2’) are a relative 
error criterion. Clearly, when the approximate optimality of x^^^ and is measnred 
by (Cl), (S.l) and (S. 2 ) are equivalent to finding (x^"''^,and (?/^+^,y^+^) such that 


r E (9())fc(x^+^), </ifc+i with Y.t=o^^k+l<oo, 

\ 7?^+^ £ «9V’fc(y^+^), < z^fc+i with Y.'k=o'^k+i<co. 


If the approximate optimality of x^^^ and y^^^ is measured by (C2) or (C2’), (S.l) and 
(S.2) are equivalent to finding (x^'*'^, ^^+^) and (y^+^, y^'*'^) such that with p = 1 or 2, 


r ^'=+1 £ a4(x^+i), < Pfc+i||x^+i-x'=||r^ with E^oMfc+i <oo, 

\ £ 5V’fc(/+^), E£o^fc+i <oO' 
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Remark 3.1 (a) When the proximal operators Vf and Vg are chosen as jdX for a constant 
/3 > 0 and the step-size r is set to be 1, the lEIDP-ADMM with (Cl) reduces to the lADMl 
in [24j- If, in addition, taking T = Q = the lEIDP-ADMM with (C2’) requires 

( e x^\\^_AA*+isx with E£o/^i+i<oo> 

\ - y^\\aBB*+px with Y!k=o^k+i 

whereas the LADM2 in [24] is actually requiring that and satisfy 

( £ d(t>k{x^+^), < Hk+i/3\\x’^^^- x^W with E^o^fc+i 

\ £ d'ilJk{y^+^), < J^k+il3\\y^^^- y^W with E^o^l+i <oo- 

Since y/P\\x^+^-x^\\aAA*+dx > /3||x^+^-x*^|| and ^/P\\y’^+^-y’^\\„BB*+|3X > 

the above inexact criterion (C2’) is looser than Criterion 2 used in [24j- 

(b) When Vf and Vg are chosen to be self-adjoint positive semidefinite operators, the 
lEIDP-ADMMs with fik = i^k = 0 reduce to the semi-proximal ADMM in [38, 9]. 

(c) For the self-adjoint positive definite linear operators T and Q in (C2) and (C2’), an 

immediate choice is T = and Q = Since XmmiTf) CLnd Aniin(7J5) are 

^minw/j '^minK 'g) J ' \ id ■' 

easy to estimate, such a choice is convenient for the numerical implementation. 


Next we study the properties of the sequence generated by the lEIDP-ADMMs. For 
convenience, we let h{x, y) := A*x + B*y — c for {x, y) £ X x Y, and for each k > 1 write 


■'e 

.k . „.k 


vt ■■= / 


y\ zl-.= z^-z*- 


Ay^ := / - /-i, Ax^ := x^ - x^-\ A/ := / - z’^-\ 

Using these notations and noting that h{x*,y*) = 0, we can rewrite Step (S.3) as 

/ = 2 ^+i-ra/i(x^+i,/+i) = /+i-rn(Xx^+i+. 8 *y^+i). ( 12 ) 

Lemma 3.1 Let \^{x^, y^, z^)^ be the sequence generated by the lEIDP-ADMMs with 
and {y^,r]^) satisfying equation (10) or (11). Suppose that Assumption 2.1 holds 
Pg also satisfies Vg + ^T^g 


and the operator Vg also satisfies Vg + ISg ^ 0. Then, for all k > 0 we have 


^+1 I — 1/* 11 ^/c+l 112 ||^/c||2^ _j_ _ 11^ 


( 2 -r)a||Mx"+',y"+')r + (ra) ||z;-|r-||z, 


I Il'T’^lP -t- HA?/' 

+ iFe \\Vf+T.f~ \\Xe\\Vf+T,f + W^y 

„k „,k\ t 2 *A„,k-\-l\ 


e I 
,fc+l||2 


fc ||2 
e HT^ 


ST" 4^9 


- llAy 


k \\2 




< 2 (l-r)/T(h(x^y'=),^*A/+^)+r'=+^- 
where r^~^^ := 2 (Xg“''^, + 2 (yg+^, + 2{'q^~^^ —rf^, Ay^"*"^). 


Proof: From the expressions of (fk and ifk and equations (10) and (11), it follows that 

^k+l _ ^ j3*yk_c) _ VfAx'^+^ £ 5/(x^+^), (13) 

^k+1 _ _ ^j3^*ggk+i ^ _c) _ VgAy’^^^ £ 9y(/+^). (14) 


7 




Substituting the first identity in (12) into equations (13) and (14) respectively yields 

(T-l)a^h(x^+\/+^)-^/+^+cr^^*A/+^ -P/Ax^+^+^^+^e5/(x''+^), 
(r-l)cri3/i(x^+\/+^) - VgAy^+^ + € dg{y'"+^). 

In view of inequalities (5) and (6), from the last two inclusions and equation (9) we have 

{x'l+^,{T-l)aAh{x^+^,y^+^)-Az^+^+aAB*Ay’^+^-VfAx^+^+^^+^)> 

{y'!+\iT-l)aBhix^+\y'^+^)-Bz^+^-VgAy^+^+y^+^) > ||ye^+^|||^. 

Adding the last two inequalities together and using equation (12) yields that 

(r-l)n||h(x^+\/+i)f-(rc7)-'(Az"+Sz^i) + n(Mx'^+\/+i),^*A/+i) (15) 

- (x^+\iP/Ax^+^-^^+^) - {y’!+\{Vg+aBB*)Ay^+^-y^+^) > \\x^e^^\\% + \\y’!+^\\%. 

Next we deal with the term a{h{x^~^^,y^~^^),B*Ay^~^^) in inequality (15). Notice that 

a(Mx"+\/+'),B*A/+i) = (l-r)a(h(x'=+\/+i)-Mx",/),B*A/+i) (16) 

+ {Az’^+\B*Ay’^+^) + (l-r)cJ(/^(x^/),i3*A/+l). 


We first bound the first two terms in (15). From equations (14) and (12), it follows that 

-Bz^+^ + (r-l)cj.B/i(x^+\/+^) - PgA/+^ + 77^+^ e dg{y’^+^), 

-Bz'^ + (r-l)ai3h(x^ /) - VgAy'^ + 77^6 dg{y^). 


Combining the last two inclusions with the second inequality in (6) yields that 

(r-l)n(Mx"+\/+') - - (Az'=+\^*A/+i) 

-(A/+i-A/,PgA/+i) + (r 7 ^+^-r 7 ^ A/+1) > ||A/+1|||^. (17) 

Using equation (8) and the given assumption Vg + |Sg ^ 0, we have that 


(A/-A/+i,P,A/ 


') = - ^I|A/Il7, - lllA/ 




- I|A/ 




< - ^iiA/iit, - 1 iiA/+-i 

< ll|A!/' 




k\\2 


-l||Ay+i||?. J,, +1||A!,‘+1|||. (18) 




where the last inequality is using ill7/^ ^11^ , 3 „ <||A7/^"^ 

"' 9 + 8^9 




+ l|Ay' 


fc+l||2 

'In- 


Combining inequalities (18) and (17) with equation (16), we immediately obtain 
cJ(/^(x^+^/+l),.B*A/+l) < (l-T)^7(/^(x^/),B*A/+l) + ( 77 ^+ 1 -77^A/+l) 




fc+l||2 




*^+1111. (19) 
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Now substituting inequality (19) into equation (15), we immediately obtain that 
(r-l)n||h(x"+\/+i)f - (ra)-i(Az"+\ 4 +') - 
+ (l-T)a(/^(x^/),5*A/+l) + 


'^9+4^9 


- {<£*', {r, + aBB‘)Ay^+') + ir*+‘ > + Hj/J+'III, + i||A/+‘|||,. (20) 

By the first equality of ( 8 ) and equation (12), the term {Az^~^^, can be written as 


{Az^+\zX) = ]^\\zXf -\\\4f + ^ 


{x^+\yXf- 


Applying equation ( 8 ) to (Xg+^,P/Aa:^+^) and ,{Vg+ aBB*)Ay^^^) yields 

{x’l*\V,Ax>‘*') = hx^+Vv, - l\\4\\v, + 1 l|A*‘+‘l&,. 


(Xg + <TBB*)Ay'^'^^) - 2 llyl^^llps+ffse* ~2^^yg\\%+< tbb- + 2 IIll^+ffoe- - 


Substituting the last three equalities into inequality (20), we have that 

(r-2)a\\h{x'^+\yXf + {Ta)-\\\zX-\\zXf) + X^^^^^^^ 

+ {X\\vf+i:f-\\xX\\vf+^f) + (Il^y 1 ln+ 3 s - \\^yX\\+iX 


+ 2{l-T)a{h{x\y^),B*AyX + r^^^-\\^yX\i 


Tb 


> \\Xe 


k+l\\2 


+ Ikelll, + IIAx^'+i^^ + ||y, 


A:+l||2 




,, fc + l ||2 


So + lll/e llso “ oil'll/ IISo 


,fc+l ||2 


Notice that + ||x^|||^ > i||Ax^+H^ and + 

The last inequality implies the desired result. The proof is completed. 


□ 


The following lemma provides an upper bound for the term in Lemma 3.1. 


Lemma 3.2 If (Cl) is used in (S.l) and (S.2), then for any given 'y > 0 we have 

|r^+iI < ^-1 X+i\\xX\\Tf+^k+i\\yX llrj + 7"^ i’^k+i + z^fc)IIA/+^ 11^^ 

+ 7 ||' 7 }“^||/Ufc +1 + 7ll'7^~^ll(2z"fc+i+^^fc) for fc > 1 ; ( 21 ) 

if the criterion (C2) is used for the minimization in (S.l) and (S.2), then 

Ir'^+^l < lZfc+i||x^+i||^^ +z/fc+i||yg^+^||^^ +/ifc+i||Ax'=+i^^ 

+ (z^fc + 3i^fc+i)||Ay^'’’^||^ + iyfc||A?/*^||^ for A: > 1; (22) 

and if (C2’) is used for the minimization in (S.l) and (S.2), then for any given 7 > 0, 

Ir'^+^l < lihlX^XllTf + XMXt,) +X\\^xX\Tf 

+ ( 27 -^ + 7 z/|+ 7 z/^+i)||A/+^||^^+ 7 -^||A/||^^ for /c > 1. (23) 
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Proof: When the criterion (Cl) is used in (S.l) and (S.2), for any given 7 > 0 we have 

(24) 

+ -f\\Tg-^\\{iyk+i+i^k) 


for all k > 1. Indeed, when Hk+i = 0, the first inequality in (24) holds since now = 0; 
and when y,k+i > 0, from equations (7) and (10) we have 


fc+l cfc+l\| ^ k-k+l ||^fc+l||2 


2 |(x^+\e''+i)| < 




7 


MTf + 


k-k+l 


WTf^wwe 


fc+l||2 ^ yk+1 ||^fc+l||2 


7 


Tf+^W'^f ^Wk'k+l- 


Similarly, we can prove that the last two inequalities hold for all A: > 1. Adding the three 
inequalities in (24) yields (21). When (C2) is used in (S.l) and (S.2), for all A: > 1 


2 |( 4 +SI < fik+M^^\\%+lik+i\\Ax 


fc+l||2 


117?’ 


< z^fc+i||ye+i| + i^fc+i||A/+i| 

|2 

\\% 


(25) 


2|(,^"+l-r?^A/+l)| <(z.fc + 2z.fc+i)||A/+i|||+z.fc||A/"2 


r„- 


Indeed, when Uk+i = 0, the second inequality in (25) holds since now = 0; and 
when Uk+i 7 ^ 0) from equation (7) and ^ Tj and Q~^ < Tg, it follows that 

2|(!/7ty+‘)| < *'/.+iii!/7‘ii7.+^ii7+’ig < •'/.+iI|97‘ii1+^mI|a/+'ii|. 

Similarly, we can prove that another two inequalities hold for all A: > 1. Summing up the 
three inequalities in (25) yields (22). When the criterion (C2’) is used, for any 7 > 0, 

2\{xl+\e^^)\ < 7/xi+i||x^+^||^^ + i||Ax'^+i|||-^, 

2|(ye^+\7'^+^)| <7i^i+il|ye+if, + ^l|A/+i^,, (26) 

2 \{ y ^+^- y \ Ay ^+^)\ < (7-1 + 71.2 + 7^.2^,) ||A/+ 1 ||^^+ 7 - 1 ||A/||^^ 

for all /c > 1. Indeed, when Vk = 0, the third inequality in (26) holds since now = 0; 

and when Vk 7 ^ Oj from equation (7) and 7 Tf and Q~^ 7 it follows that 

2|{,‘+‘-,‘,A/+i)| < (7^| + l7+l)||A/+‘||2_. + 7j||,‘||2+-^||,‘+i||| 

7^fc l^k+i 

< (7"^ + 7z^l + 7i^i+i)l|A/+l^^+7-iA/||^^. 

Similarly, one can prove that another two inequalities hold for all A: > 1. Summing up 
the three inequalities in (26) yields (23). The proof is completed. □ 


Based on the results of Lemmas 3.1 and 3.2, we obtain the following proposition. 
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Proposition 3.1 Let {{x^,y^, z^)}k>i be the sequence generated by the lEIPD-ADMMs. 
Suppose that Assumption 2.1 holds and the operator Vg also satisfies Vg + ^ 0. 

(a) If the criterion (Cl) is used in (S.l) and (S.2), then for any given 7 > 0 ?re have 




7 

.. k \\2 


2 /ifc+i + t^fc+i 
7 




k\\2 


'g~<~4^S " ' "’^9 + 1^9 


+ {2-t -2j-^Hk+i)cr\\h{x^^^,y’^"^^)\\^ 

< 2(l-T)o-(/i(x'',/),i3*A/+^)- (l- 7 "^z^fc+i- 7 "^i^fc)||A/+^||^^ 
-||Ax^+^||J^^i^^+7||7}"^||/ife+i+7|IVi(2i/fc+i + i^fc) for k>l. 

(b) If the criterion (C2) is used for (S.l) and (S.2), then we have 

{'^—hk+l)\\Xe~^^\\pj.^Y.f~\\^e\\pf+Ef + {^ — ^k+1 — ^hk+l)\\ye~^^\\rg ~ Wv^"'^ 


e WTg 

k ,.A:mi 2 


+ + ||Ay'=+^||J,^^3s^- ||Ay' 


< 2(l-r)cj(/i(a;^,/),^*A/+^) - (l-z^fc-4i/fc+i-4/Xfc+i)||A?/' 


k\\2 

^9 + § ^9 
fc+l||2 


WTg 


_ II A+ l ||2 

" "77+is^-/.fc+i(77+sA 


for A: > 1. 


(27) 


(c) If the criterion (C2’) is used for (S.l) and (S.2), then for any given 7 > 0 we have 

{'^- 7 hl+l)\\Xe'^^\\Vf+I^f-\\Xe\\vf+^ + (l- 7 Z^i+l -37^1+1) bebe llr^ 
+ {xa)-^[\\z’f'^^f-\\zlf)+ (2-r - 2 . 57 ”^ - 1.57/i|+i)o-||/i(x'=+\ 

+ W^y'^Xg^li^g - +7-bA/+if, -7-^l|A/b^ 

_( 1 _ 67 - 1 - 7 z .2 + 47 -Vb(x ^/)||2 for k > 1 . (28) 

Proof: (a) From inequality (21) and the result of Lemma 3.1, it follows that 

1 f||^^^+l|| 2 _||^fc|| 2 ') + ^l_!(^MU.fc+l|| 2 _ n..fc „2 

TfJ 


(2-T).j||ft(i‘+‘./+‘)ll=+— (Il 4 +>lp-ll 4 lp)+(i— m)||„; 

|ufc+l||2 

7 


+ (l“^^7“) \\xX\\%+T.j-\\xt\\vf+T.f + \\^V^^^\\%„^?Ly.„-\\^y^"‘^ 


9^ 4^9 


'^9~^4^9 


% 


< 2(l-r)a(h(x^ /),^*A/+1)-1|Ax"+i||- (i_||A/+bi2 

+ \\xX\\lAA*~^^\\'Tf ^bfe+i+7ll'^ ^||(2i^fc+i+z^fc)- 

Since XXWIaa* A (^\\hX^\yX\\^ + \\yX\\lBB* A (T\\h{x’^+\yXf + \\yX\\Tg 
where the second inequality is due to aBB* A %, the last inequality implies part (a). 
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(b) From inequality (22) and the result of Lemma 3.1, it follows that 


(2-T)a||ft(l‘+‘,/+')||" + (T<r)-1 (114+11|2- ||4||2) + 

~ lll/ell^ + (l“l^fc+l)||a^e'''^llpy.+S/“l|3^ellp/+Sj' + ^k+lW^U^'^^WTg 

< 2{l-T)a{h{x\y%B*{y^+^-y^)) - ||A/+1||^^ 


For the terms llXg"*"^ ||_ 4 _ 4 * and \\Ax’^+^\\aa-, using equation (7) yields that 


ll-^e WAA* 

= 

||h(x'=+i, 


+ 

WVe^^l 

|2 

Wbb* ~ 

2(/i(x''+\ 

yk+l 

).s*9L') 



< 

||h(x"+^ 


+ 

l|ye"+'l 

|2 -L 

\bb* + 


yk+1 

)II"+4||4+' 

is*! (30) 

l|2 

l\x Waa* 

= 

||h(x^+^ 


+ 

\\h{x^ 


+ 11 A /+1 

l|2 

Wbb* 





-2(/i(x^+S/+^; 

),h{x^,y 

^))+2(h(x^/) 

-h{x^+\y^+^),B*Ay^+^) 


< 

||h(x^+^ 


+ 

\\h{x\ 


+ IIA/+1 

l|2 

Wbb* 

+ jll'>T+‘, 

yfe + l)||2 


+ 4||ft(i*,y)l|" + l|fc(i‘,/)|l" + l|A/+i|||e. 

+ l|IM+i‘+‘./+‘)ll" + 2||A/+i|||e.. (31) 

Combining the last inequalities with (29) and using aBB* :< Tg yields part (b). 

The proof of Part (c) is similar to that of part (b), we here omit it. □ 

4 Convergence analysis of the lEIDP-ADMMs 

In this section we analyze the convergence of the lEIPD-ADMMs with the approximation 
criterion (Cl) and (C 2 ) respectively chosen for the minimization in (S.l) and (S. 2 ). 

4.1 Convergence of the lEIDP-ADMM with (Cl) 

For convenience, we write := (Xg; y^\z^; Ay^; Az^) for /c > 1, and let "H : X x Y x Z x 
YxZ—>-XxYxZxYxZ denote the block diagonal operator defined by 

i^:=Diag((P/ + S;)V2, (r,)V2, {Vg + l'Lgf/\ ^x) 

with the proximal operators Vj and Vg satisfying Pj + Sj ^ 0 and 7^ + |Sg ^ 0 . 

Lemma 4.1 Let {(x^, z^)}fc>i he the sequence generated by the lEIDP-ADMM with 
the criterion (Cl) and ma'x.{jjk,i^k) A for some constant 7 > 0. Suppose 
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that Assumption 2.1 holds and Vf and Vg also satisfy Vf + ^Tij ^ 0 and Vg + ^T^g ^ 0. 
Then, when r G (O, 2), there exists an absolute constant c > 0 such that for all k > 1 


\\nw^ 


A:+l||2 ^ 

\m+i - 


1 + 


4:Vk 


7 ( 2 -r)J L 


1 + 




7 


Ww'^Wwk + C7(i"fc+ I^k+1+ Tk+i) 


— mm 


- min(T, l+r-r^), min(T, l+r-r^)] + II , 

where Wk :XxYxZxYxZ—)-XxYxZxYxZ is the block diagonal linear operator 
Wfc := Diag(^(l-7"Vfc)2^, ^k-'y~^’^k)d:, I, I, (2-r-27"Vfc)2^)• 

Proof: For each letVfc:XxYxZxYxZ^XxYxZxYxZbe defined by 


Vfc = Diagfx, I, I, I, ^ ^ — l) with ak = ^k + Vk-i- 
V 1-7 ^ak 1 


7 

With the notations Wfc and Vfc, we first establish the following important inequality: 

ttfc+l' 


< — min(r, 1 + r —r 


T 


1-7 lofc+i 
+ max(||7}"^||,2||7^"^||)7(afc+i+/rfc+i). 


a;^/)|P+(l-: 


7 


l|Ay^ 


fc+l ||2 


(32) 


Indeed, when r G (0,1], since 7 = 7 < 1) by eqnation (7) we have 


2 (l-r)a(h(x^/),^*A/+^)< 


fc+l\ ^ ,,fcM|2 


1 -7 lOfc+l 




Substituting this inequality into Proposition 3.1(a) and using (12), we obtain that 

/ \ II fc+l ||2 II fc||2 I / -I ^fc+1 II fc-i-l ||2 II A 

^ jlla^e \\Vf+i:f-\\Xe\\Vf+i:f + [^- ^ ^ Wr.-WVe 

^) + ||Ay^+bi2 


.fc ||2 


+ 


rcT 






7 7 

_||A/h2 


1^9+4^9 


+ 2-t- 


‘^Tk+l\ 1 


7 7 r^cr 


( 2 -r)(T 


1 


Az 


fc ||2 


< 


< 


1-7 lofc+i r^cr 

1 -T-L., ii'‘Y./)iy (i-^)iiAy-^-iii._i._,).BB 


-||Ai*+'||^^^,j,^+ 7||T, ■||m+i+7||T, '|l(2Fi+, +vt) 


- - Si -||/.(x*,/)l|"-r(l-^)||As 

1-7 '-Uk+i ^ 7 ^ 

+ max(||r^"i||,2||7^"i||)7(afc+i+yfc+i), 


fc+l ||2 
iTg 


(33) 
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where the second inequality is using Tg — ^ rTg and Vj + ^ 0. For the 

case where r G (1, 2), from + Vk) < 1 and equation (7) we have 


2 (l-T)a(/^(x^/),B*A/+l) 


< 


cj(r-l) 


r(l - 7 ^ak+i) 




+ t{1 — 


«fc+i 

7 




Substituting this inequality into Proposition 3.1(a) and using (12) yields that 


+ iT(T) 


^ jll^e lb/+S/ IbellPz+Sj- , ^ ^ 

-lni..fc+l||2 _ ||^fc||2^ _g (||A?/^+^b II A„.fc||2 


b, 

+ {2-r - ?»tl)^||A.'=+l||2 - ,^71- 

7 ^ T^(j 1—7 ^at._Li 




Is ll^/llj, + |sj 


k \\\2 


7 


7 «fc+i 


1-7 ^afc+i 


«fc+i 

7 




< - 


- Il^a; + ||^^^i^^+7||7 \\nk+i+l\\Tg ||(2i/fc+i + z/^) 
(1 + T“^ —t)iT 


■llft(i‘./)lP-(i + T-7{i-2m)||A!, 


,fc+l ||2 


1-7 iafc +1 

+ max(||7“^||,2||7“^||)7(afc+i+/ifc+i), 


(34) 


where the second inequality is using Vj + ^ 0 and Tg — {T—l)TaBB* ^ (1+r—r^)7- 

Notice that Wk >- 0 and Vfc 7 0 for all /c > 1 due to max(/rfc, Vk) < 7 ™in(^, ^^). From 
the definition of and the expressions of 71, Wk and V^, the left hand side of (33) 

and (34) equals ll^'W^^llv^.+i' Along with (33) and (34), we get (32). 

Now by the assumption max(/xfc, Uk) < 7 min(i, 7^), it is not difficult to verify that 


1 


2 _ life _ ‘^k'k 
7 7 


< 1 + and 


1 


< ( 1 +^^) 


7 


(l_lA^)(l_^i^) 


7 


1 + 


4t'fc 


7(2 - t)J' 


By the expressions of V^+i and Wfc, we have Vfc+iW^ ^ ^ (1+ 


\\nw'^\\l,^,= {nw\Vk+iW^^Wk'Hw'^) < ( 1 + 


1 + 


7 

dz'fc 


1 + 


7(2-r) 


X. Then 


7(2 - t)J 


ll^^^llvv,- (35) 


In addition, since max(^fc,i/fc) < 7 min(^, 7^) /c > 1, we have | < 1 —7^ — 

Let c = max(||7~^||j 2||7~^||)- Combining (35) with (32) yields the desired result. □ 


By Lemma 4.1, we may establish the convergence of the lEIDP-ADMM with (Cl). 

Theorem 4.1 Let {(x^, z^)}k>i be the sequenee generated by the lEIDP-ADMM with 

(Cl) and max(/rfc,z'fc) < 7 min(^, 7^)? where 7 is same as that of Lemma f.l. Suppose 
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that Assumption 2.1 holds and Vf and Vg also satisfy Vf + ^ 0 and Vg + |Sg ^ 

0, Then, for (a) T £ (0. or (b) T € (i^,2) but E£o + 

I|A!/‘+'II5;) < oo, the sequence {{x^,y^)} converges to an optimal solution of ( 1 ) and 
the sequence {z^} converges to an optimal solution to the dual problem of (1). 

Proof: We write idk = [I + ^( 2 -r) ][l+ = c-fivkA- i^k+i + li-k+i) and 

Rk = ^\\h{x^,y^)\\'^ + \\Ay^~^^\\‘^^ for k >1. By Lemma 4.1 we have that 

2 k—l k 

pw^^’^^llvvfc+i <-min -min(T, l+r-r^),mm(r,l+T-T2) ^ JJ idjRi + Rk^ 

1=0 j=l+l 

k k—l k 

j=0 1=0 j=l+l 

Since Ylh=i ^'k <oo, we have 1 < fl^i some Ki > 1 . Hence, we have that 


^i(P^°ll>Vo + Ef=o^ 0 -( 0 ,^^), 

^ Ki{\\nw^\\^^ +EzLo^0 + Ki{T^-l-T)Y^’f^QR^ if r G [^^,2). 


Notice that niax(/rfc, z/fc) < oo implies Yl’i^o'^i ^ ^2 for some K 2 > 0. Then, 


I min(r, 1+t-t") < Zfl{l|Ww»||Jv. + ^ 2 ) if T £ (0, i^), 


< f<'l(ll«i»”llvv. + r 2 )+Ki{r^--l-r)j:l„R‘ if r £ |i^,2) 


--T— implies Wk A W for some W 0. Then, un- 

2 max(3,23:^} 


Notice that max(/xfc,i^fc) < 

“ 2-T ! 

der conditions (a) and (b), the last two inequalities imply that the sequence {Rw^'\ is 
bounded and Yl^o ^ +oo. The latter implies that lim^^oo Rk = 0, and consequently. 


lim ||Ay^’''^||7- = 0 and lim ||Az^|| = 0. 


k^oo 


k^oo 


(36) 


By equality (12), the limits in (36) imply that the sequence {||irg'’'^||^yi*} is bounded. By 
the definition of R and the boundedness of {Rw^f, the sequence {||a;g+^ H-p^+s^} is also 
bounded. Thus, the sequence Ht^} is bounded. In addition, the boundedness of 

{Rw^'\ also implies that the sequences {||ye'*'^||75} and {||2;e'’'^||} are bounded. Together 
with the positive definiteness of Tf and If, it follows that {{x^,y^,z^)'\ is bounded. So, 
there exists a convergent subsequence, to say {(x^, y^, z^)}yc- Without loss of generality, 
we assume {{x^,y^, z^)}]c —)• (x°°, y°°, z°°). Since lim^^oo = 0, we have 

h{x°°,y°°) = 0. In addition, taking the limit k ^ oo with k £ JC on the both sides of 
(13) and (14), and using the closedness of the graphs of df and dg (see [27]), we have 

-A*z°° £ df{x°°) and -B*z’^ £ dg{y^). 

Along with (9), (x°°, y°°) is an optimal solution of (1) and 2:°° is the associated multiplier. 
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Finally, we argue that {x°°, y°°, z°°) is actually the unique limit point of {(x^, z^)}. 

Recall that is an optimal solution to (1) and is the associated multiplier. 

Hence, we could replace (x*,y*, z*) with {x°°,y°°, z°°) in the previous arguments, start¬ 
ing from (13) and (14). Thus, inequality (32) still holds with (x*,y*, z*) replaced by 
{x°°,y°°, z°°). Hence, from the definition of max(/rfc, i/^) < oo and equation 

(36), we have limfc^oo.fceAC < limfc^oo,fceK; W'Hw'^W^ = 0 where 

W = Diag(^X, I, I, I, (2 - t)x). 

This means that for any e > 0, there exists a sufficiently large ko € JC such that 
\\T~(-W^°\\WkQ < 2^ ^ 27 ^- Lemma 4.1, for any k > ko we have 

W'Hw'^+^W^ < n + 

j=ko l=ko j=ko-\-l 

k 

l=ko 

This, by the positive definiteness of W, shows that lim^^oo = 0. Consequently, 

lim ||x^+^-x°°||p,+Ef = 0, lim ||/+^-2/°°||r, = 0 and lim ||z'=+^-z°°|| = 0. 

k—^oo ^ ^ k—^oo k^oo 

Combining limfc^oo “2/°°||rg = 0 with Tg >- 0 yields lim^^oo 112/^"'"^ —2/°° 11 = 0. In 
addition, the second limit in (36) implies lim/j^oo h{x^,y^) = 0. Together with 

+2||h(x"+i,/+i)f, 

we obtain \mik^^\\x^~^^ — x°°\\‘j^j^t, = 0. Noting that Y\m.k^^\\x^~^^ — x°°\\'Pjj^Y,f = 0, 
we have lim^^oo —x°°||^ = 0. By the positive definiteness of 7/, it follows that 

limfc^oo ||x^-x°°|| = 0. Thus, limfc^ooa;^ = 3:°°, lim^^oo 2/^ = 2/°° and lim^^oo= '2°°- 
That is, {x°°,y°°, z°°) is the unique limit point of {{x^,y^,z^)}. □ 

Remark 4.1 Theorem 4-1 shows that one can establish the convergence o/{(x^, z^)} 

generated by the lEIDP-ADMM with (Cl) ifVf and Vg are chosen such that 

Tf + ttLIj ^ 0, Tg —Tig P 0, Tj Tf <7.4.4* 0, Tg Tg cBB* P 0. 

2 o 

In fact, using the same arguments, one can get the convergence of {{x^, y^, z^)} generated 
by the lEIDP-ADMM with (Cl) ifTf and Tg are chosen such that for some a G [^, 1), 

Tfp—Tf ^ 0, TgP—Tg ^ 0, TfTTfPcjAjA* i- 0, aTgTTgPcrBB* 0. 
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4.2 Convergence of the lEIDP-ADMM with (C2) 

For each /c > 1, we write := zj^; Ay^; Ay^; Az^), and let4^:XxYxZxYx 

YxZ^XxYxZxYxYxZbe the block diagonal linear operator defined by 


i^:=Diag((iP/ + Sy)i/2, (r,)V2^ 



X, (iP, + |Sg)i/2, 



for the proximal operators Vj and Vg satisfying Vf + Y^f Y 0 and Vg+jYg Y 0. To establish 
the convergence of the lEIDP-ADMM with (C2), we need the following lemma. 

Lemma 4.2 Let {{x^,y^ , z^)}k>i be the sequence given by the lEIDP-ADMM with (C2) 
and max(^fc, Vk) < min(0.1, Suppose that Assumption 2.1 holds and Vf and Vg also 

satisfy Vf + Y 0 and Vg + Y 0. Then, when r G (0, 2), for all k>l we have 

< [1+ 10i/fc+40(//fc+i+i/fc+i)] [l+ + 2/rfc+i) 


2-r 




lOcT, 




+ \\Ay 


fc+l||2 

lx 


— min(0.1 min(r, 1+ r— r^), min(T, 1+ r— 
where the operator Wk :XxYxZxYxYxZ—>-XxYxZxYxYxZfs defined by 
Wfc := Diag((l-(1-5/rfc- I, I, {2-t -3y,k)T). 

Proof: Let Vfc:XxYxZxYxYxZ—^-XxYxZxYxYxZ for k > Ihe defined by 

2-t^ 

O-k 

With the notations Wk and Vfc, we first establish the following important inequality: 
,,fc+l||2 ll'7/,.,fc||2 


Vk = W\a,g{x, X, X, X, Uk-iX, ^ + 6/rfc)X^ with ak = fk-i + ^{i^k + hk)- 




< — min(T, 1 + T —T^) 


a 


+ (l-aA:+i)||Ay^ 


A:+l||2 

lx 


(37) 


_r(l-afc+i) 

Indeed, when r G (0,1], since otk+i < 0.9 by max{ykWk) < 0.1, it follows from (7) that 

k g*A„.fc+l\ ^ Wkfr^k ,,fcM|2 _L n .MIA,X+1||2 


2{1-T)a{h{x ,y ),B*Ay‘ 


■\\h{x\y^)r + {l-ak+,)\\Ay^^%-r).BB* 


1—«fc+i 

Substituting the last inequality into Proposition 3.1(b) and using (12), we obtain that 
(l-;Ufc+i)||a:^+^||^^+s^-||a:^||^^+Sj- + {f^-W+i - 5/ifc+i) 


e \\Tn 


+ 


-l/|Ufc+l||2 


- Z, 


fc ||2 


) + l|Ay' 


fc+l||2 




- l|Ay 


k\\2 




+ z^fc+i||A/+^||^^ - z^fc||A/||f^ + (2-r - 3/ifc+i)cr||/i(x^+\/+^)|p 
/ 2-t „ ^ 1 

“ W- 


< - 


'g -II ^ ii/g 

-h Gfj.k+i)—^\\Az^\\‘^ 

afc+i 

(t||/i(x^,?/^)|P 


T^a 


1 —afc+i 


(l “fc+l) l|Ay^’'’^||xg-(l-r)(TBB* 


< - 


a 


1 —afc+i 


||Mx^/)f-T(l-a,+l)||A/+l||^, 


(38) 
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where the last inequality is using Vf + ^T,f—fj,kj^i{Vf+'Ef ) ^ 0 and Tg — (l—T)aBB* ^ rTg, 
implied by Vf + |S/ ^ 0 and ^k+i < 0.1. When r G (1, 2), by (7) and max(/Xfc, Uk) < 0.1, 


2(l-rMMx^/),^*(/+l-/))< 


cr||/i(x^,/)||2 r-1 


^ + (l-afc+OII^/^iaCr-WBB- 
Substituting the last inequality into Proposition 3.1(b) and using (12), we obtain that 


(l —/J-fc+l) — \\Xe\\pf+T,f + (1“Z^fc+1 “ 5/ifc+l) lli/e^^ll^ — Wu. 


.k\\2 




+ z^fc+i||A/+^||f^ -i/fc||A/||^^ + (2-r-3/rfc+i)CT||/i(x''+\/+^) 

-(r 




2_^ + 6/rfc+i)^||A2"f 
-ak+i T^cr 


^ (1+T ^-T)a||h(x^/)f . AIIAw'^+MP 

^ (J- afc + ljll^y \\Tg-{T-l)TCTBB* 


1 —afc+1 
- ||Ax^+^"^ 


< - 


'Pf+h^f -Mfe+i (Pf+^f ) 
(1+ T~^- T)(T||/i(x^,y^)|p 


- (1 + r - T^)(l-afc+i) ||A/+^| 


iTg- 


(39) 


1 —afc+i 

where the last inequality is due to Vf + — fik+i{Vf + Sj) A 0 by /ifc+i < 0.1, and 

Tg - {T-l)TaBB* A (1+T- r^)7^. 

By the definitions of the vector and the operators 'H,Wk and Vfc, the left hand side 

of (38) and (39) is — Along with (38) and (39), we get (37). 

Since 0 < max(^fc, Uk) + min(0.1, ^^) for all /c > 1, it is not difficult to check that 
< 1+ 10i>'fc+40(/ifc+i+z^fc+i) and -57;— < 1 + 


1 — Ctk+l 


1_ 

^ 2-r 


2-r’ 


which in turn implies that -- ^ J < [l+ 10i^fc+40(/rA:+i+t'A:+i)] (l+ and 

(1 <y.k^i)[l j 

2-r-3;'i;, < ■ Together with the expression of Vfc+iW^^, we obtain that 

Vk+iW^^ + [ 1 + Wuk+ 40{fCk+i + i^k+i)] ( 1 + ( 40 ) 

Since Vk < 0.1 for all A: > 1, we have 0.1 <1— Ok+i < 1- Now combining (40) with 
inequality (37) yields the desired result. Thus, we complete the proof. □ 

By Lemma 4.2 one may obtain the following convergence result of the lEIDP-ADMM 
with the criterion (C2). Since the proof is similar to that of Theorem 4.1, we omit it. 
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Theorem 4.2 Let {{x^,y^, z^)}k>i be the sequenee generated by the lEIDP-ADMM with 
the criterion (C2) and max(/xfc, < min(0.1, Suppose that Assumption 2.1 holds 

and the operators Vf and Vg also satisfy "Pj + ^ 0 and ^ 0. Then, for 

(a) r G (0,i^) or (b) r G [i^,2) but Er=o + II< oo, 

the sequence {{x^,y^)} converges to an optimal solution of problem (1) and the sequence 
{z^} converges to an optimal solution to the dual problem of ( 1 ). 

Remark 4.2 Theorem 4-2 shows that one can establish the convergenee of {{x^,y^, z^)} 
generated by the lEIDP-ADMM with (C2) ifVf and Vg are chosen such that 

Vf + 'z'Pif ^ 0, Vg + 'pEg P 0, Tf + Vf + crAlAl* P 0, Tig + Vg + cBB* P 0. 

o o 

In fact, using the same arguments, one can get the convergence of {{x^, y^, z^)} generated 
by the lEIDP-ADMM with (C2)ifVf andVg are chosen such that for some ai, 02 G [^,1), 

Vf-\-—Tf P 0, Vg-\-—Tg P 0, aiTfVVfPo'A.A* P 0, afTgVVgPcrBB* P 0. 

4.3 Convergence of the lEIDP-ADMM with (C2’) 

Let for A: > 1 be same as the one of the last subsection. Define the block diagonal 
linear operator P:XxYxZxYxYxZ—;>XxYxZxYxYxZby 

P:=Diag((Py+S;)V2, (r,)i/2^ (Ta)-% {Vg + ^Tg^/^ {Tg)^/^ (rV)"'^) 

with the proximal operators Vf and Vg satisfying Vf + Tf P 0 and Vg + jTg P 0. 

Lemma 4.3 Let {{x^,y^, z^)}k>i be the sequence generated by the lEIDP-ADMM with 
(C2’) and max(/i|,r'|) < min(^, —) for some eonstant 7 > 360. Suppose that 

Assumption 2.1 holds and the operators Vf and Vg also satisfy Vf + ^ 0 and Vg + 

^ 0. Then, when r G (0, 2), the following inequality holds for all k >1 

f -^^min(T,2.6-1.6T)||/i(x'',2/^)|p-ci||A/+i||^^ if r G (0,1-6] 
y +2^|2.6-1.6T|||/i(x^,?/^)|P+C2||Ay^+^||^^ if rG (1.6,2) 

where ci = 2 -t-6.57-^ - , C 2 = | 2 -t- 6 . 57 -i - |, 

and Wfe :XxYxZxYxYxZ^XxYxZxYxYxZis the block diagonal operator 

Wk ■= Diag((l-7/i^)X, (l-7i/| -37/r^)X, X, X, 7-^X, (2-r-2.57"^-1.57/i^)X). 
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Proof: Let V:XxYxZxYxYxZ—)-XxYxZxYxYxZbe the block diagonal 
linear operator defined by V := Diag(X, X, X, X, X, (2—r—2.57“^)X). With the notations 
Wfc and V, we first establish the following inequality 

Ilk+I - 11 ^^" llv < 2 . 6 - 1 . 6 r) [l- 67 "'+ vl^^)] || ||r, 

-Cfc(7,r)||Mx^/)||^ (41) 


where 


6.5 max(l-r, (r-l)/1.6) 
CkKl.T) := 2-r- 


7 1-67 1-7(1^1+ z^|+i)' 

Indeed, when r G (0,1], from 1—67“^—7(z^| + > 0 and equation (7) it follows that 

cj(I-t) 


2a(l-r)|(Mx^/),^*A/+l)| <- 




--67 ^+ T ^ l + i ) 

+ [l-67-^-7(i/| + 

Substituting the last inequality into Proposition 3.1(c) then yields that 


- piw^Wl + Cfc(7, T)(T\\h{x^ ,y^)f 

< [1-67-1- 7 (z/| + ^l+l)] l|A/+l||r,-(l-r)aBB* 

< -t( 1 - 67 - 1 - 7 i/|- 7z/|+i)||A/+i||r,, (42) 


where the last inequality is using Pj + iSj- —7 ^{Vf + Tif) ^ 0 and Tg — {l—r)aBB* ^ rTg- 
When r G (1,2), from 1— 67-1 — 7(z^| + z^|+i) > 0 and equation (7) it follows that 


1.6(1-67 i-7i^fc-7z^fc+i) 

+ (l-67"^-7z^|-7i^fc+i)l|A/+i||^6(r-iVBB*- 
Substituting it into Proposition 3.1(c) and using the notations Wfc and V, we have 

llk+i - + Cfc(7, r)a||/r(x^ /) IP 

< [1-67-1-7(1/! + z/l+i)] ||A/+i||!-^_i,6(,_i),eg* 

< -(2.6- 1.6t)(1-67-1-7z/!-7i/!+i)||A/+1||^^, (43) 


where the last inequality is using Pj + — 7 ^{Vj + Sj) ^ 0 and Tg — 1.6{T—l)aBB* ^ 

(1 —1.6(r— 1))7^. From (42) and (43), we immediately obtain inequality (41). 


Now by the given condition max(/i|, z/|) < min(g. 


2 ,,2^ ^ 2-t-2.57~^ \ 1 


)-, we can check that 


2 -r- 2.57 


-1 


2 —T —2.57 ^ — 


2 — 


< 1 + 


37/^1 


2-T-2.57 


-1 


and 


1 


1 - 7z^! - 371^! 


< 1 + 2'y{i'l + 3fil). 
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Together with the expressions of V and Wfc, it is not difficult to verify that 

^ max (^1 + 1 + 27(t'i + 3/ri))x. 

Combining this relation with (41) and the condition max(/i|, v‘f.) < min(|, —)i, 

we obtain the desired result. The proof is completed. □ 

By Lemma 4.3 we can establish the following convergence result of the lEIDP-ADMM 
with the criterion (C2’). Since the proof is similar to that of Theorem 4.1, we omit it. 

Theorem 4.3 Let {(x^, z^)}k>i be the sequenee generated by the lEIDP-ADMM with 

the criterion (C2’) and max(/x|,i^|) < min(^, —) for some eonstant 7 > 360. 

Suppose that Assumption 2.1 holds and Vj and Vg also satisfy Vf + 7 0 and Vg + 

0. Then, for (a) t e (0,1.6] or (b) r G (1.6,2) but (|2.6-1.6r| ||/i(x^, y^)|p+ 

C2||A/+ifJ < 00, the sequence {(x^,y^)} eonverges to an optimal solution of (1) and 
the sequence {z^} eonverges to an optimal solution to the dual problem of (1). 

To close this section, we want to point out that the convergence of the inexact positive 
definite proximal ADMM [24] with (Cl) and a special (C2’) is only established for r = 1, 
while the convergence results of Theorem 4.1 and Theorem 4.3 extend it to the inexact 
indefinite proximal ADMM with r € (0, '^ 2 ^ ) and r G (0,1.6], respectively. 

5 Applications to doubly nonnegative SDPs 

Let Sf be the cone of n x n positive semidefinite matrices in the vector space of n x n 
real symmetric matrices, endowed with the Frobenius inner product (•, •) and its induced 
norm || • ||. The doubly nonnegative SDP problem is described as follows: 

max I-(C, A) | AeX = bE, AjX >bi, X £ 5”, X-M G /c| (44) 

where Ae '■ ^ and : S" —)■ are the linear operators, bE £ and 

bj £ are the given vectors, and X — M £ X means that every entry of A — M is 
nonnegative. We always assume that Ae is surjective. The dual of (44) has the form 

min [5^r^i{yi)-{bi,yi)) + {6jc*{Z)-{M,Z)) - {bE,yE) + bs^iS) 

s.t. Ajyj -£ Z -£ AeVe S = C (45) 

where X* is the positive dual cone of X. For the four-block separable convex minimization 
problem (45), one may use the multi-block ADMM with Gaussian back substitution 
[16, 17] or the proximal ADMM [31] to solve. In this section, we apply the lEIDP- 
ADMMs for (45) by viewing {yj,Z) as a block and {yE,S) as a block (respectively. 


21 






viewing {Z^ue) as a block and S as a block when m/ = 0). Notice that, by introducing 
a slack variable, problem (44) can be equivalently written as 

max|-(C,X) I AEX = bE, AiX-x = bi, X X-M gK, x > o|, (46) 

and an elementary calculation yields the dual problem of (46) as follows 

min-{bi,yi) + Z)) - {bE,yE) +5<S!f('S') 

s.t. Ajyi Z A*EyE -\- S = C, yj — z = 0. (47) 

Problem (47) is still a four-block separable convex minimization since {z, Z) can be solved 
simultaneously. Hence, in this section we also apply the lEIDP-ADMMs for solving (47) 
by viewing (y/, z, Z) as a block and (y^:, S) as a block. 

Throughout this section, instead of using the constraint qualification (CQ) in As¬ 
sumption 2.1, we use the following more familiar Slater’s CQ for problem (45): 

Assumption 5.1 (a) For problem (44), there exists a point X G such that 

AeX = bE, AiX >bi, X G int (5"), X G X. 

(b) For problem (45), there exists a point (jjj, Z, yE, S) x S” x x S” such that 

AJyi Z A*EyE S = c, Z G X* , yj G S G int (5”). 

By [2, Corollary 5.3.6], under Assumption 5.1, the strong duality for (44) and (45) holds, 
and the following Karush-Kuhn-Tucker (KKT) condition has nonempty solutions: 

AeX — bE = 0, 

A*jyj Z A*EyE S — C = 0, 

< (A, S) =0, X gSI, S G SI, (48) 

{X,Z) =0, X gX, Z gX*, 

, {yi,AiX-bi) = 0, AiX-bi >0, yi G 


5.1 Numerical results for the DNNSDPs without AjX > bj 

In this case since the linear operator [X Ag]*[X A*^] is not positive definite, we impose 
a semi-proximal term ^{Z —Z^, yE —y^)Diag(eX, 0)(A —Z^, yE —y^Y' guarantee that 


Tf ^ a[Z A*e]*[X A*e] + Diag(eX,0) = a 


A*e 

Ae AeA*e 


^0, 


(49) 


and propose the following partial lEIDP-ADMM for problem (45) with three blocks, 
where for a given a > 0, the augmented Lagrangian function of (45) is defined as 

La{yi,Z,yE,S,X) := {S^rnj[yj) - {bi,yi)) + {dic*iZ) - {M,Z)) - {bE,yE) + Ss"{S) 

+ {X, A}yi + Z+A*EyE-\-S— C) + —1| A/yj-l-Z +A^EyE~\~S— CH 

V(y/, Z, yE, S, X) G x x x S" x 
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Algorithm 5.1 (A partial lEIDP-ADMM for (45) with three blocks) 

(S.O) Let T = qX — AiA*j for g > Let cr, r > 0 be given. Choose a 

small constant e > 0 and a point (Z^, y^, S^,X^) = (0,0, 0,0). Set k := 0. 

(S.l) Compute the following problems by one of the criteria (Cl) and (C2): 

Z,yE ^ 

= IlsAC-A*Ey%^^-Z^+^-a-^X^). 

s 

(5.3) Update the Lagrange multiplier X^~^^ via the following formula 

^k+i = + r(7(Z^+i + A*Ey^E^^ + 5^+1 - C); 

(5.4) Let /c •(— /c + 1, and go to Step (S.l). 


For the approximate optimal solution (Z^^^,y^^) of subproblem (50), one may get 
it by solving the problem minz^y^ 4>k{Z,yE) in an alternating way. Let /cq = k. The 
iterates {Z^CVe) yielded by solving the problem min^^y^ (j)k{Z,yE) alternately satisfy 

= argmin()fc(Z, y^^-i) and y^^ = argmin()>fc(Z''L 74 .) for j = 1 , 2 ,... . 
ze §" ve&^^e 


From the expression of the function (pki','): h is immediate to obtain that 

f 0 G AfK*{Z^^) -M + X^ + cj(Z^i+A)j7fe'"'+S^-C) + e{Z^i-Z^), 
[ 0 = AeX’^ -bE + aAE{A*EyE^+Z>^^+S'^-C). 


Comparing this system with the optimality condition of TcCnz^y^ (f)k{Z,yE)^ with = 

(yA*E{yE^ —yE^~^) we have (^^■’,0) G dcfkiZ^UyE^)- This means that {Z^CUe^) satisfies 
the criterion (Cl) with = 0 when ||^^^ || < yk+i and '^’^=QUk+i < 00 . In addition, let 

5 := (Vo- + e-v^)min (^a+s, --jZ=\min{AEA*E)^. 

By using equation (49) and [20, Theorem 7.7.6], it is not difficult to verify that 


TfhfJ 


/«T+£ — 


\/ cr+£:X 


0 


Vcr+g- 


/a+e 


^AeA% 


SI. 


This means that .^yj) satisfies the criterion (C2) with T = S ^I and Uf^ = 0 once 




Ve^W"^ + ^WZ^^-Z^W^ and Y(,T=oUk+i < oo. 
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since the right hand side of the first inequality is less than —Z^] yE-yE)\\Tf 

In the seqnel, we call Algorithm 5.1 with the snbproblems in (50) solved alternately by 
the criteria (Cl) and (C2) lEIDP-ADMMl and IEIDP-ADMM2, respectively. 

We apply lEIDP-ADMMl and IEIDP-ADMM2 for the donbly nonnegative SDPs 
without inequality constraint AjX > bj, and compare their performance with that of 
the 3-block ADMM of step-size r = 1.618 (for short, ADMM3d). Among others, the 
doubly nonnegative SDP test examples can be found in [31]. We have implemented 
lEIDP-ADMMl, IEIDP-ADMM2 and ADMM3d in MATLAB, where e = 10“^ and 

= min(0.1, ^1,001 ) for k > 1 are used for lEIDP-ADMMl and IEIDP-ADMM2. Notice 
that when \\{Z^^ — Z^■, — y^)\\-jj < 1, the criterion (C2) is more restrictive than (Cl). 

Moreover, the criterion (C2) will require much more inner iterations as the primal and 
dual infeasibility becomes smaller since \\{Z^^ — Z^] y^^ — y^)\\'jj is close to 0. So, in the 
implementation of IEIDP-ADMM2, we modify the criterion (C2) into 

< max(\/5/rfc+i||(Z^^-Z^;y^J-?/^)||7^,0.1max(r/p,r/z))), (51) 

where rjp and yp are defined below. In addition, the implementation of ADMM3d here 
is different from that of [33] since the former uses the solution order Z —>• —)• S', while 

the latter uses the order —)■ Z —)• S. The computational results for all DNNSDPs are 
obtained on a Windows system with Intel(R) Core(TM) i3-2120 CPU@3.30GHz. 

We measure the accuracy of an approximate optimal solution {Z,yE, S,X) for (44) 
and (45) by using the relative residual y = max {yp, yo^ys^yK^ ys* j yK* j yCi ; yc 2 } where 

WApX-hEW \\A*EyE + S+Z-C\\ I|n5?(-A)|| ||nyc*(-X)|| 

1 +115^11 ’ =-TT^-’ 1 + llAll ’ = 1 + llAll ’ 

I|n55(-S)|| ^ ||n^.(-z)|| ^ (A,S) (A,Z) 

1 + ||S|| ’ 1 + ||Z|| ’ 1 +||A|| + ||S||’ 1 +||A|| + ||Z||' 

We terminated the three solvers lEIDP-ADMMl, IEIDP-ADMM2 and ADMM3d when¬ 
ever y < 10“® or the number of iteration is over fcmax = 20000. 

In the implementation of the three solvers, the penalty parameter a is dynamically 
adjusted according to the progress of the algorithms, and the idea to adjust a is to balance 
the progress of primal feasibilities {yp,ys^yic) dual feasibilities {yD,ys* ^yic*)- The 
exact details on the adjustment strategies are not given here. In addition, all the solvers 
also adopt some kind of restart strategies to ameliorate slow convergence. During the 
testing, we use the same adjustment strategy of a and restart strategy for all the solvers. 

Figure 1 shows the performance profiles of lEIDP-ADMMl, IEIDP-ADMM2 and 
ADMM3d in terms of number of iterations and computing time, respectively, for the total 
605 (including BIQ(165), RCP(120), 0-|_(113), FAP(13) and QAP(95)) tested problems. 
We recall that a point (x, y) is in the performance profiles curve of a method if and 
only if it can solve (100y)% of all tested problems no slower than x times of any other 
methods. We see that lEIDP-ADMMl and ADMM3d need the comparable iterations and 


24 











computing time. Among others, IEIDP-ADMM2 requires the least number of iterations 
for 60% test problems, but it needs the most computing time which is about 1.5 times 
that of lEIDP-ADMMl and ADMM3d for about 80% test problems. 


Performance Profile (iter) (total 506 test problems) q = 10 ® Performance Profile (time) (total 506 test problems) r) = 10 ® 



Figure 1: Performance profiles of the number of iterations and computing time of solvers 


5.2 Numerical results for the DNNSDPs with AiX > bj 

For this case, we may apply the proposed lEIDP-ADMMs for solving (45) or (47). Firstly, 
we report the numerical results of the lEIDP-ADMMs for solving problem (45). 


5.2.1 Numerical results of the lEIDP-ADMMs for problem (45) 

Since [X A*j\* [X A*i\ and X]* [A*^ X] are not positive definite and Ai is not surjective, 
we introduce the semi-proximal terms \{yi —yf, Z —Z^)Diag((TT, 0)(y7 — y^, Z — Z^)"^ 
and ^{ys —y%i S — S'^)Diag(0, eX)(yE — S to guarantee that 


Tf 


ha[A*i Z\*[A*i X]+Diag(£7r,0) 




(52) 


and 


Tg'^alAl X] + Diag(0,eX) = ct 


AeA*^ 

A% 


Ae 

£+ex 


^0, 


(53) 


and propose the following partial inexact indefinite proximal ADMMs for solving (45). 
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Algorithm 5.2 (A.n inexact indefinite-proximal ADMM for (45 ) ) 

(S.O) Let T = qX — AiA*j for g > Amax(-4/A)). Let cr, r > 0 be given. Choose a small 
constant e > 0 and a point {y^, Z^, y^, S^,X^) = (0,0, 0,0, 0). Set k := 0. 

(S.l) Compute the following problems by one of the criteria (C1)-(C2): 

« argmin(/)fc(2//,^):= L^{yi,Z,y%,S^,X^) +\\\yi-y'f\\lT (54) 
{yl+\S'^+^)^^TgrninMyE,S)-.=L^{y^+\z'^+\yE,S,X^)+\\\S-S'^\\l^. 

yE,s ^ 

(5.3) Update the Lagrange multiplier X^~^^ via the formula 

A^+i = + rcj(A^y^+^ + + 5^+^ - C). 

(5.4) Let fe ■(— A: + 1, and go to Step (S.l). 


One may obtain the approximate optimal solutions (yj"*~^, and {y^^, by 

computing lainyj^z (t>k{yI, Z) and miny^^s f’kiyE, S) in an alternating way. Let ko = k. 
The iterates (yj^Z^J) for j > 1 yielded by minimizing (f>k(yj,Z) alternately satisfy 

I 0 G (y^^)-bi + AiX^ + aAiiAJyj^ + + A^yf, +5^= -O) + aT{yj^-y^), 

\ 0 G A/)c* {Z^^ ) - M + + a{A*jy^^ + +A%y'f; + S’^-C). 

Let = aAi{Z^i—Z^^-^). Comparing the last system with the optimality condition of 
minyj^z ((kiyi, Z), we have (^^■’ ,0) G d(j)k{l/i^ iZ^^). This means that ,Z^i) satisfies 
the criterion (Cl) when < pik+i and Y(jV=oh''k+i < cc. Notice that 




'gX 

Ai 


X O' 

-1 

1- 

o 

H 

X -g-^Ai 

A*j 

X 


1 

1 

* 


0 X-g-^A*jAi 

0 X 


1 -1 


^min(£i, Amin(2^-£' A*jAi)) 


-1 

H 

o 

-1 

H 

1 

1 

1 

1 

* 


0 X 






X 

— q~^A 


0 


-1 A* X 


= idXr 


X -g-^Ai 


'X -g-^Ai 
0 X 


X 0 

-g-^A*j X 


-w 


-1 


X^idX 


where dt := min (^>, 1—g ^Amax(Al)A/)). So, {y^^, Z^i) satisfies (C2) with F = -^X when 
U’^^W < /ifc+iV^^||y^ -y^ + A)(z^. -Z^)||2 + -yf P - ||A)(Z^.-Z^)||2. 
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k ■ u 

The iterates {Ve^S^^) for j> 1 yielded by minimizing il^kiVE^S) alternately satisfy 

r 0 = -bE + AEX^ + aAE{A*jy^+^ + Z^+^+A%yE^+S^^-^-C), 

\ 0£ A/'55(S^0 + + A*EyE^ + S^^-C) + £{S^i - S^). 

Let = aAE{S^^ —S^i-^). Comparing the last system with the optimality condition of 
problem miny^^s'^kiVE, S), we have (T]^i,0) £ dipkiys^, S^^). This means that {yE^ ,S^i) 
satisfies (Cl) when ||7?^^|| < t'fc+i with ^ addition, let 

6 := {y/a + £ ^_ Amin(-4E-Ag), Vo'+e ) • 

V,/cj+e / 

By using equation (56) and [20, Theorem 7.7.6], it is not difficult to verify that 


Tg^a 


\/gH-e— 


/cr+e 


^AeAX 


0 


^ 5Z. 


0 

This means that {yE^ ,S^^) satisfies the criterion (C2) with Q = 5~^X once 

||r?'^^'|| < ^Ek+i^WAliy^A-y%) + (5^^ - 5^)P + 


We call Algorithm 5.2 with the two subproblems in (S.l) solved alternately by the criteria 
(Cl) and (C2) lEIDP-ADMMl and IEIDP-ADMM2, respectively. 

We apply the lEIDP-ADMMl and IEIDP-ADMM2 for solving the extended BIQ 
problems described in Section 4.2 of [31], and compare its performance with the four- 
block proximal ADMM of step-size r = 1.618 (although without convergent guarantee) 
by adding a proximal term f jjy/ — y/||^ for the yj part, where T = — 

We call this method PADMM4d. The computational results for all the extended BIQ 
problems are obtained on the same desktop computer as before. 

We measure the accuracy of an approximate optimal solution {X,yj, Z,yE, S) for 
(44) and (45) by the relative residual rj = max {'np,r]D,'ils,VlC, VS*, ViC*, VCi, VC 2 ,'ni, Vl*}, 
where rjp-,rjs-,'nK.^'ns*-,TlK.*-,flCi^'nC 2 defined as before, and £ 10 , 111 ,Vl* are given by 

_\\A*jyi+Z + A*^yE+s -C\\ _ || max(0, 6/- ^ 7 A)|| _ || max(0,-y 7 )|| 

1 +lie'll 1+ 116/11 l + ||y/|| • 

The three solvers lEIDP-ADMMl and IEIDP-ADMM2 and PADMM4d were stopped 
whenever r] < 10“® or the number of iteration is over /Cmax = 40000. 


Figure 2 plots the performance profiles of lEIDP-ADMMl, IEIDP-ADMM2 and 
PADMM4d in terms of the number of iterations and computing time, respectively, for 
the total 165 extended BIQ tested problems. It can be seen from this figure that lEIDP- 
ADMMl, IEIDP-ADMM2 and PADMM4d are comparable in terms of the iterations and 
computing time, lEIDP-ADMMl and IEIDP-ADMM2 need the least number of itera¬ 
tions for at least 80% tested problems, which is about 90% that of PADMM4d, and 
PADMM4d requires the least computing time for about 70% tested problems, which is 
about 90% that of IEIDP-ADMM2. 
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PerformancB Profile (iter) (165 Extended BIQ problems) t] = lO'® Performance Profile (time) (165 Extended BIQ problems) r| = 10" 



Figure 2: Performance profiles of the number of iterations and computing time for EBIQ 


5.2.2 Numerical results of the lEIDP-ADMMs for problem (47) 

Since [A} 0 0 X] + [X — X 0]*[X —X 0] and [A*^ X]*[A*^ X] are not positive 

definite, we introduce the semi-proximal terms z—z^, Z—Z^)T>iag{aeT, 0,0)(y/ — 

z — z^,Z — Z^Y' and \{yE —2/ei S — 5^)Diag(0, eX){yE — y%, S —S^Y to ensure that 


Tfh(T([A*i 0 X]*[A*j 0 X] + [X -X 0]*[X -X 0]) + Diag(o7eX,0,0) ^ 0 (55) 


and 


Tgha[A*E X]*[A*e X]+Diag(0,eX) = (t 


AeA*e Ae 


A*r 


£+£X 


^ 0 , 


(56) 


and propose the following inexact indefinite proximal ADMMs for solving (47), where 
for a given cr > 0, the augmented Lagrangian function of problem (47) is defined as: 


La{yi,z,Z,yE,S,X,x) := -{bi,yi) -h (^;) -b {5k.* (Z) - {M, Z)) - {bE,yE) 

+ <^5" (<5) + {X, A*jyi-\-Z -\-A*EyE + S — C) + {x,yi — z) 

'^\\-^}yi~^^~^-^*EyE+S—C\\ +—||y/ —^11^ (57) 

V(y/, z, Z, yE, S, X, x) £ x x S’" x M”"® x S’" x S" x R"""". 
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Algorithm 5.3 (A.n inexact indefinite proximal ADMM for (47 ) ) 

(S.O) Let a,T > 0 be given. Choose a suffieiently small constant £ > 0 and an initial 
point {y^, z^, Z^, S^,X^,x^) = (0,0,0,0,0,0,0). Set k := 0. 

(S.l) Compute the following problems by one of the criteria (C1)-(C2): 

[y^+\ « argminL<,(y,"+\ z’^+\ Z>^+\yE, S, X^x'^) 

yE,s ^ 

(5.3) Update the Lagrange multipliers via the following formula 

X^+i = X^ + Ta{Z^+^ + A*iv’^+^ + A*Ey^+^ + - C), 

^k+i ra{y^+^ - z^+^). (58) 

(5.4) Let k k + 1, and go to Step (S.l). 


For the approximate optimal solution , z^^^, in (S.l), one may get it by 

solving the problem vaAiy^^z^z 4>k{yiZ^ Z) in an alternating way, where 

4(2/7, := 4(2/7, 42/1, S^X^x’^) + ^||2/7-2/f f • 

The iterates {y^\ z^C given by solving rmiiiyj^z,z (t>k{yiZ: alternately satisfy 

y^i = aigm.m.f)k{yEZ^^-\Z^^-^), {z^\Z^i) = argmin (fkiy^Cz^Z) forj = l,2,... 
yeK^i {z,Z)£W^ixS'^ 


with kQ = k. We apply the conjugate gradient method to the first minimization, i.e., 

X^' 


{AiA*MI+£)X) = 


z^^-^+Ai{c-Z^i-^-S’^-A*Ey% - —) 


X —hi k 

+ £y’f 


a 


+B^\ 


where denotes the error yielded by the conjugate gradient method. Let 

= a{Z^i - Z^i-^) + a{z’^i - /^-i) - aR^C 

Then, together with the definition of {z^RZ^^), we have (^*^^,0,0) € d()>k{yf\z^CZ^^)- 
This means that {y^\ z^R Z^^) satisfies (Cl) when \\f,^^\\ < y-k+i and YlT=oyk+i < oo. 
For the approximate optimal solution {y^^, in (S.l), one may obtain it by solving 

the corresponding minimization alternately. Also, from Subsection 5.2.1 it follows that 
{ys^S^^) satisfies the criterion (C2) with G = 5~^Z if r/^7 = cjAe(5^j — satisfies 

||?7^^|| < V6auk+i^\\A*^{yE^-y'f;) + (5^j - S'^)P + -S'^p. 
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We call Algorithm 5.3 with the subproblems solved alternately by (Cl) lEIDP-ADMMl. 

We apply the lEIDP-ADMMl for solving the extended BIQ problems described in 
Section 4.2 of [31], and compare its performance with the previous PADMM4d and the 
four-block ADMM of step-size r = 1.618 (although without convergent guarantee). We 
call the latter ADMM4d. The computational results for all the extended BIQ problems 
are obtained on the same desktop computer as before. We measure the accuracy of an 
approximate optimal solution (X, yj, z, Z^ue, S) for (44) and (47) by the relative residual 
T] = maxwhere r]p,r]s,r]ic,VS*,VlC*,r]Ci,VC 2 
are defined as before. The solvers lEIDP-ADMMl and PADMM4d and ADMM4d were 
terminated whenever i] < 10“® or the number of iteration is over fcmax = 40000. 


Performance Profile (iter) (165 Extended BIQ problems)r| = 10 ® Performance Profile (time) (165 Extended BIQ problems)!! = 10 ® 



Figure 3: Performance profiles of the number of iterations and computing time for EBIQ 

Figure 3 plots the performance profiles of lEIDP-ADMMl, PADMM4d and ADMM4d 
in terms of the number of iterations and computing time, respectively, for the total 165 
extended BIQ tested problems. We see that, when applying the lEIDP-ADMM for 
solving the dual problem (47), the number of iterations and the computing time of the 
lEIDP-ADMMl are still comparable with those of ADMM4d, but PADMM4d requires 
more 4 times iterations than lEIDP-ADMMl and ADMM4d as do for at least 80% 
test problems. This means that a small proximal term as possible is the key to the 
performance of proximal-type ADMMs. The computing time of PADMM4d is a little 
less than that of lEIDP-ADMMl and ADMM4d since the latter solves an m/ x mj linear 
system with the conjugate gradient method, where mj may attain 374250. 
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6 Conclusion 


We developed an inexact indefinite proximal ADMM of step-size r £ (0, '^ 2 ^ ) with two 
easily implementable inexactness criteria for the two-block separable convex minimization 
problems with linear constraints, for which it is either impossible or too expensive to 
obtain the exact solutions of the subproblems involved in the proximal ADMM. Numerical 
results for the DNNSDPs with many linear equality and/or inequality constraints show 
that the inexact indefinite proximal ADMMs are effective for this class of difficult three 
or four block separable separable convex optimization problems with linear constraints. 
Among others, the inexact indefinite proximal ADMM with the absolute error criterion 
(Cl) is comparable with the directly extended ADMM of step-size r = 1.618, whether 
in terms of the number of iterations or computing time, and is superior to the one with 
the relative error criterion (C2) by weighing the number of iterations and the computing 
time since the latter is very restrictive and requires too many iterations for the solution 
of subproblems. In our future research work, we will explore other easily implementable 
inexact criteria like relaxing fj-k+i and Vk+i in (C2) to be a constant, and study the 
nonergodic convergence [6, 7] for the inexact indefinite proximal ADMMs. 
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